Multimodal Integration - A Statistical

|This paper presents a statistical approach to developing multimodal recognition systems and, in particular, to integrating the posterior probabilities of parallel input signals involved in the multimodal system. We rst identify the primary factors that innuence multimodal recognition performance by evaluating the multimodal recognition probabilities. We then develop two techniques, an estimate approach and a learning approach, which are designed to optimize accurate recognition during the multimodal integration process. We evaluate these methods using Quickset, a speech/gesture multimodal system, and report evaluation results based on an empirical corpus collected with Quick-set. From an architectural perspective, the integration technique presented here ooers enhanced robustness. It also is premised on more realistic assumptions than previous mul-timodal systems using semantic fusion. From a methodolog-ical standpoint, the evaluation techniques that we describe provide a valuable tool for evaluating multimodal systems.